Repository navigation
Conversation
When the platform container is recreated its hostname changes, leaving runs bound to a host that no longer exists. For runs that execute on a runner the checkout lives on the runner, not the platform; the platform-side directory only stages prompts and artifacts. The stale host must not block such a run forever. Distinguish platform-owned from runner-owned checkouts: - workspaceBlocked no longer treats a dead recorded host as blocking when the run has an execution runner; the runner keeps its checkout. - workspaceLost only reports a checkout as lost for platform-owned runs (no execution runner), so cloud runs keep their dead-host protection. - adoptForeignWorkspace lets a shared-workspace runner run continue on its runner instead of reporting the workspace as lost. Safety is preserved by the existing dispatch rules: once nodes have changed code, a run refuses to move to another machine (validateExecutionReconnect, bindPoolRunner), so continuing on the original runner cannot drop unpushed work. Live foreign hosts and same-host other-root binds still block takeover. Related to Tencent#1079.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(engine): keep runner-owned checkouts across control-plane restarts
This is the runner owner recovery slice of #1079. When the platform container is recreated its hostname changes, leaving runs bound to a host that no longer exists. For runs that execute on a runner, the checkout lives on the runner — the platform-side directory only stages prompts and artifacts — so a stale platform host must not block the run forever.
Problem
workspaceBlockedtreated a dead recorded host as blocking whenever the run had a default execution runner. But a runner-owned checkout survives a control-plane restart (the runner keeps its mounts), so such a run was stuck: its approval gate could not be approved and its workspace was reported as lost even though no work was actually lost.Change
Distinguish platform-owned from runner-owned checkouts:
internal/engine/run_recovery.goworkspaceBlockedno longer returns blocking for a dead recorded host when the run has an execution runner; the runner keeps its checkout.workspaceLostonly reports a checkout as lost for platform-owned runs (defaultExecutionRunner == ""), so cloud runs keep their existing dead-host protection.internal/engine/run_environment.goadoptForeignWorkspacelets a shared-workspace runner run continue on its runner instead of reporting the workspace as lost with the dead host.Safety is preserved by the existing dispatch rules: once nodes have changed code, a run refuses to move to another machine (
validateExecutionReconnect,bindPoolRunner), so continuing on the original runner cannot drop unpushed work. Live foreign hosts and same-host other-root binds still block takeover.Verification (Linux/amd64, Docker
python:3.12)internal/engine/workspace_runner_owner_test.go:TestRunnerGateRecoversAfterPlatformHostChange— approving the gate after the platform host changed succeeds; the run continues on its runner and the unpushed checkout files are kept byte-for-byte.TestRunnerWorkspaceRebindsWithoutReportingLoss— the next node rebinds the platform staging without reporting the workspace as lost.TestRunnerGateStaysBlockedByLiveForeignHost/TestRunnerGateStaysBlockedBySameHostOtherRoot— live foreign / same-host other-root binds stay blocking.TestRunnerReconnectKeepsTheOriginalRunner— a runner run never moves to another machine while the original is unreachable.TestCloudRunStillTreatsDeadHostAsLost— platform-owned checkouts keep their dead-host protection.gate MCP binaryfailures); no new failure beyond baseline.go vetand public-content checks pass.Scope / notes
TraceID == Run.IDcontract or to telemetry attribution.Related
Closes part of #1079 (runner owner recovery); does not close the whole issue.